slurm: batch the queries and parallelize the writes in the GPU power/clock helpers - #1393

…clock helpers

set_gpu_power_levels.sh and set_gpu_clocks.sh called nvidia-smi once per
GPU to read the target value and once more to apply it, all serially. Both
"nvidia-smi -pl" and "nvidia-smi -ac" take roughly a second per GPU, so on
an 8-GPU node the two helpers together add about 8 s to the prolog of every
job that 50-exclusive-gpu runs for. srun reports this as:

  srun: Prolog hung on node <node>

Read the values for all GPUs in a single --query-gpu call, then apply them
in parallel and collect each child's exit status so a failure on any GPU
still fails the script.

Behaviour is otherwise unchanged: the same values are written to the same
GPUs. The "default" branch of set_gpu_clocks.sh already operated on all
GPUs at once and is untouched.

Observed on DGX OS 7.5.0 (8x B300), Slurm 26.05.1: prolog took 6-8 s per
job while 50-exclusive-gpu was running.

Signed-off-by: Jea-Eok-Kim <je.kim@xiilab.com>

`readarray -t limits < <(nvidia-smi ...)` hides the query's exit status from
both readarray and `set -e`. The helpers therefore reported success in every
failure mode: an empty result made the write loop run zero times, a truncated
result configured only some of the GPUs, and a nonzero exit was not seen at all.

The query result now goes through a file so its status can be checked, the index
is selected alongside the values so a write targets the GPU nvidia-smi reported
instead of an array subscript, and every row is parsed and range-checked before
the first write is launched. The row count is compared against `nvidia-smi -L`,
so a partial result is rejected rather than silently applied.

set_gpu_clocks.sh additionally selected clocks.max.mem and clocks.max.sm in two
separate queries. If the two returned different row counts, `${maxMEM[$i]}` was
empty for the trailing GPUs and produced an `-ac ,1980` argument. Both values
now come from the same query, so they cannot drift out of alignment.

Verified on a DGX B300 (Ubuntu 24.04, bash 5.2) with an nvidia-smi stub that
honours --query-gpu and records writes instead of performing them. Identical
results for both helpers:

  case        before                        after
  ----------  ----------------------------  ---------------------
  ok          rc=0  8 writes                rc=0  8 writes
  empty       rc=0  0 writes                rc=1  0 writes
  partial     rc=0  3 writes (of 8 GPUs)    rc=1  0 writes
  fail        rc=0  0 writes                rc=1  0 writes
  nonnumeric  rc=0  8 writes ("N/A" passed) rc=1  0 writes

One limitation of the stub is worth stating: it records writes rather than
performing them, so the `nonnumeric` row shows 8 writes for the old code. On
real hardware `nvidia-smi -pl N/A` fails and `wait` would surface rc=1 — but
only after eight bad invocations. The new code rejects the row before the first
one.

The `ok` case is unchanged, so the parallel-write speedup this branch adds is
preserved.